USING ELECTRONIC AUDIENCE RESPONSE SYSTEMS IN HIGH SCHOOL 

AGRISCIENCE COURSES 


John W. Conoley, Assistant Teaching Professor 
East Carolina University 
D. Barry Croom, Associate Professor 
Gary E. Moore, Professor 
James L. Flowers, Professor 
North Carolina State University 


Abstract 

The purpose of this research was to determine whether using an audience response system 
improves student achievement. Audience response systems are computer-based tools for use in 
classrooms for the purpose of providing feedback to students on questions asked during 
instruction. This study used a quasi-experimental design to determine if students receiving 
feedback through an audience response system had higher achievement scores than those who 
receive feedback through non-technology based methods. The experiment included students from 
three high school agriscience classes. Teachers integrated the audience response system into the 
classroom and students ’ tests scores were examined comparing the technology-based feedback 
methods with the verbal and written response-contingent feedback methods. The results of the 
study showed a significant increase in student achievement when integrating an audience 
response system. The study concludes that audience response systems are a promising, 
developing technology for improving student achievement and positively impacting the 
classroom environment. 


Introduction 

Effective teaching can be described as 
the process of setting instructional goals, 
conducting a series of processes to 
accomplish these goals, and assessing how 
the goals are accomplished. Feedback is an 
important part of the assessment phase 
because it provides students with 
information on their performance and 
suggestions for improvement. Feedback 
significantly improves student learning and 
provides a useful connection between the 
teacher and the learner (Bruning, Schraw, & 
Ronning, 1999; Cohen, 1985; King & 
Young, 2002). 

Effective feedback provides learners 
with two types of information: verification 
and elaboration. Verification tells the 
learner if their answer to a question or 
problem was correct. Elaboration explains 
to the learner why their answer is correct or 
incorrect (Kulhavy & Stock, 1989). There 
are several methods to provide both 
verification and elaboration resulting in 
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effective feedback in the classroom, 
including written feedback, verbal feedback, 
and peer-group discussion feedback 
(Merrill, 1987). The development of 
technology and computer-based methods has 
changed the capabilities of instructional 
feedback. Computers allow instructors to 
collect and analyze large feedback data sets. 
Database software makes cataloguing 
feedback responses on different timelines 
feasible and efficient. Audience response 
systems allow instructors to instantly assess 
learning outcomes (Chiu & Woods, 2002). 

Audience response systems are a group 
of emerging technologies that are being 
incorporated into classrooms and other 
training venues. Audience response systems 
typically include a wireless, infrared 
transmitter, a receiver connected to a laptop 
computer loaded with a software package 
specifically designed for use with the 
receiver. The teacher projects a question 
onto a television or LCD projection screen. 
Then students keyed in their answer on their 
wireless transmitter. The receiver processes 
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the answers as correct or incorrect, then 
displays the correct answer while 
simultaneously storing the individual 
answers in a database for that questioning 
session. The software allows the teacher 
flexibility and many questioning, storing, 
display, and data analysis options, but the 
overall data transfer process remains the 
same. 

This study focused on a primary need of 
determining the impacts of an audience 
response system on student achievement and 
the classroom environment, and the need to 
test audience response systems prior to 
implementation in agriscience courses. This 
research project contributes to the wider 
body of research surrounding audience 
response systems by analyzing the 
applications and impacts of audience 
response systems specific to a high school 
agriscience classroom environment. 

Purpose 

The purpose of this research was to 
determine whether using an audience 
response system improves student 
achievement. Specifically, the research 
question was: Do students receiving 

feedback through an audience response 
system have higher achievement scores than 
those who receive feedback through non- 
technology based methods? The null 
hypothesis was: 

Ho: There is no difference in student 

achievement between students who received 
feedback through the audience response 
system and students who received feedback 
through non-technology based methods. 

Review of Literature 

Duncan and Biddle (1974) suggested 
that the study of classroom teaching and 
learning involves four categories of 
variables: presage, context, process and 
product. Presage variables are those 
variables associated with the teacher, and 
are typically identified as personality traits, 
teaching skills, and teaching styles (Duncan 
& Biddle). Context variables are those 
variables not influenced by the teacher. 
These variables are attributed to the 
students, the school and the community and 


are often referred to as learner variables. 
Process variables refer to the teaching and 
learning processes occurring in the 
classroom. The fourth category, product 
variables, describe the outcomes of the 
learning process such as achievement and 
degree completion (Duncan & Biddle). 

Duncan and Biddle’s (1974) model also 
established a consistent framework for all of 
the agriscience applications classes. 
Establishing a consistent framework 
contributes to this research by helping to 
control and test the presage and process 
variables and attribute any discovered 
significant results to the use of the audience 
response system technology. Establishing a 
framework also provides structure for 
researching integration of technology-based 
learning tools into the classroom 
environment. Standardization of a 
classroom environment allows for a more 
accurate comparison of traditional feedback 
methods and technology-based feedback 
methods. Two research studies that 
examined the interaction of variables in a 
classroom environment incorporated 
Duncan and Biddle’s theories were Ball, 
Dyer, and Garton (2001) and Cruikshank 
(1990). In these studies the classroom 
model is used as a framework to standardize 
the classroom environment. 

Madeline Hunter classified instructional 
feedback as simply checking for 
understanding (Hunter, 1982). Hunter 
described the process as detennination of 
whether students understand the material 
before proceeding. Hunter also noted that 
the teacher must know that students 
understand before proceeding to practice. 
This is the process of collecting feedback, 
analyzing the feedback, and making an 
assessment of whether or not learning has 
occurred. The instructor’s role in feedback 
intervention is to intentionally adjust and 
improve student behavior through 
instructional communications (King & 
Young, 2002). Effective communication 
between student and instructor begins with 
the instructor. The instructor must first 
collect, store, analyze and evaluate student 
responses and behaviors. 

The ability for instructors to process 
feedback information is limited by the 
boundaries of human cognition. The 
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teacher’s ability to remember and make 
sense of the data being received through 
interaction with students can be explained 
by Miller’s infonnation processing theory. 
Miller (1956) theorized that short-term 
memory could only hold five to nine chu nk s 
of information. A chunk could refer to any 
information that is meaningful. An example 
of infonnation divided into chu nk s is a 
telephone number such as 123-456-7890. 
The concept of chunking became a basic 
element of information processing theory. 
Miller also found that processing new 
information involved gathering, storing and 
representing infonnation. Miller categorizes 
these elements as encoding, retention, and 
retrieval. Instructors are able to encode, 
retain, and retrieve infonnation and are able 
to effectively provide feedback intervention 
in an instructional environment (Miller; 
King & Young, 2002). Unless instructors 
can adequately and effectively encode, 
retain, and retrieve feedback information, 
they cannot significantly influence the 
learning process (Chandler & Sweller, 1996; 
Miller). 

Eight commonly used levels of feedback 
have emerged as a standard for 
non-technology based feedback 
interventions: no-feedback, knowledge-of- 
response, answer-until-correct, knowledge- 
of-correct-response, topic-contingent, 

response-contingent, bug-related, and 
attribute-isolation (Kulhavy & Stock, 1989; 
Merrill, 1987). An aggregate score based on 
the number of correct answers with no 
reference to individual questions is the no- 
feedback intervention. Knowledge-of- 
response feedback indicates whether 
individual answers are correct or incorrect. 
Answer-until-correct feedback requires the 
learner to continue to answer until the 
correct answer is selected. Knowledge-of- 
correct-response feedback provides item 
verification and identifies the correct 
response to each item (Kulhavy & Stock). 
Topic-contingent feedback verifies 
correctness of response or provides 
additional information that will assists 
students in finding the correct answer if the 
student response is incorrect. Response- 
contingent feedback provides the learner 
with an explanation of why the incorrect 
answer was wrong and why the correct 
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answer is correct (Kulhavy & Stock). Bug- 
related feedback provides learners with 
information on specific errors and references 
“bug libraries” or collections of corrections 
to a variety of common student errors. 
Finally attribute-isolation feedback provides 
learners with item verification and 
information that focuses the learner on the 
central concept to improve the general 
understanding of the error (Kulhavy & 
Stock). 

Written and verbal or audio feedback 
delivery methods are perhaps the most 
commonly used varieties of feedback 
delivery (Cohen, 1985; Kluger & DeNisi, 
1996; Merrill, 1987). Wunsch (1982) found 
no differences between written and verbal 
feedback delivery methods. Wunsch tested 
for differences by providing both written 
and group oral feedback with students 
developing business letter writing aptitude 
and found both to be effective. Moore 
(1977) found no differences in achievement, 
but did find that audio feedback had positive 
effects on the classroom environment. 
Moore found in a study of university 
students that the students who received the 
audio feedback were more pleased with it 
than were those who received the written 
feedback and that slightly less time was 
required to provide audio feedback than 
written feedback. DaRosa, Mazur, and 
Markus (1982) found that written feedback 
was more effective than verbal feedback 
delivery, and that within the area 

of non-technology based feedback, a highly 
structured feedback delivery method was 
more effective than feedback delivery 
methods that were less structured. 

Another aspect of feedback that is often 
disputed by researchers is a comparison 
between the effects of immediate feedback 
and the effects of delayed feedback. Sturges 
(1978) conducted a study with 
undergraduates to test the effects of 
immediate vs. delayed feedback on retention 
of items on a computer-managed test. The 
results of the study showed a significantly 
higher rate of retention in the two groups 
that received the delayed feedback. Smith & 
Wight (1988) found that students favored 
immediate feedback delivery methods, and 
were more enthusiastic in their assessment 
of the technique and believed that it helped 
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facilitate learning. DiBattista, Mitterer, 
& Gosse (2004) also found that students 
favored immediate feedback when presented 
with a multiple-choice assessment. The 
immediate feedback technique used in this 
study was an answer-until-correct feedback 
technique. Students believed that the 
immediate feedback was more enjoyable, 
contributed to their learning and they 
indicated a strong desire to have immediate 
feedback for all of the prescribed multiple- 
choice tests. DiBattista et al. concluded that 
the immediate feedback delivery process 
had a broad appeal to a variety of students. 
King, Young, and Behnke (2000) found that 
the use of a combination of both immediate 
and delayed feedback was the most effective 
approach to learning. This study indicated 
that immediate feedback intervention was 
more effective when automatic processing 
occurred, while delayed feedback produces 
greater change with tasks involving 
deliberative and effortful processing (King 
et al.). Clariana (1992) found that those who 
received delayed feedback exhibited more 
retention for items that were easier, while 
immediate feedback had more impact on 
those questions that were more difficult. 

During the past few decades, technology 
has increasingly been used to implement 
feedback systems (Kulik & Kulik, 1988; 
Mory, 1994; Sturges, 1978; Zappe, Sonak, 
Hunter, & Suen, 2002). There are a wide 
variety of technologies currently being used 
to implement feedback systems. Some 
examples of these technologies are web- 
based feedback systems, video-graphical 
feedback systems, and hand-held wireless 
feedback systems. All of these feedback 
systems can be used in a variety of settings, 
but according to other feedback research, 
each of these systems will work best for 
particular educational settings (Kluger & 
DeNisi, 1996). 

There are documented advantages and 
disadvantages for using technology-based 
feedback. The ability to provide immediate 
feedback on all student responses is a 
primary advantage to using technology- 
based feedback (Mason & Bruning, 2003). 
Mason and Bruning also state that the most 
important outcomes of technology-based 
feedback are identifying errors, becoming 
aware of misconceptions and motivating 
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further learning. Some other positive 
aspects of technology-based feedback, as 
described by Mason and Bruning are that 
computers can tirelessly provide feedback 
that is unbiased, accurate, and 
nonjudgmental. Technology-based feedback 
can also be customized for learning styles of 
individual students, which is a learning goal 
that is difficult to achieve through traditional 
feedback implementation vehicles. 
Audience response systems are relatively 
new as a pedagogical feedback tools. The 
precursors of today’s audience response 
systems have been in use since 1976 with 
the first of these systems being permanently 
mounted hard-wired systems. The original 
development of a wireless audience 
response system as a pedagogical tool 
originated from the effective use of 
precursor hard-wired systems and from the 
success of multimedia use in the classroom 
(Abrahamson, 1999). A disadvantage of a 
technology-based feedback system is that 
the use of these systems has been mainly 
focused on feedback for discrete responses. 

Several studies have detennined that 
technology based feedback did not influence 
students' achievement (Merrill, 1987; Mory, 
1994). More research describes technology- 
based feedback as tools that enhance 
learning and positively affect student 
achievement (Clariana, 1992; Clariana & 
Lee, 2001; Clariana, Wagner, & Rohrer- 
Murphy, 2000; Morrison, Ross, 
Gopalakrishnan, & Casey, 1995; Pridemore 
& Klein, 1991; Waldrop, Justen, & Adams, 
1986; Whyte, Karolick, Neilsen, Elder & 
Hawley, 1995). The discrepancies in these 
research findings indicate that particular 
delivery methods of feedback are most 
effective in specific learning environments. 
The goal of this research was to find out 
whether technology-based feedback delivery 
methods or non-technology based delivery 
methods were more effective in the 
agriscience learning environment. 

Although most researchers agree that 
feedback interventions are useful, Kluger 
and DeNisi (1996) found while feedback 
increased perfonnance on average, more 
than 33% of feedback interventions were not 
effective or actually decreased learning. 
Feedback can be effective in one scenario 
and not effective in another (Cameron & 
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Cotrell, 1970; Quigley & Nyquist, 1992). 
Kluger and DeNisi proposed that feedback 
interventions related to meta-task processes 
or those such as treats or praise actually 
inhibit performance; while feedback is 
centered on task motivation or the process of 
learning enhance student achievement. 

Methodology 

This study used a post-test only, quasi- 
experimental design. This design addressed 
the differences in student achievement for 
those students using the audience response 
system compared to those students using 
more traditional feedback delivery methods, 
such as verbal and written response- 
contingent feedback. This included testing a 
total of three sections of Agriscience 
Applications, taught by three different 
teachers at Southern Nash High School. 
Achievement data was collected in all three 
sections, from two instructional units in each 
section. Two classes served as the treatment 
groups and the other class served as a 
comparison group, for the first instructional 
unit. Then during the second instructional 
unit, the group roles were switched. This 
resulted in a modified switching replications 
design. The classes were purposefully 
assigned which classes would receive the 


treatment for each lesson to balance the 
groups for comparison. A true switching 
replications design was intended to be 
implemented, requiring four intact groups. 
However, two class sections were combined, 
creating only three intact groups. 

Preliminary steps during this part of the 
design included training the three teachers in 
the use of the audience response system 
technology. Two days of three hour per day 
periods of training, facilitated by the 
researcher, took place. Also the two 
instructional unit tested were developed 
based on standard competencies and were 
tested for both validity and reliability. 

As each instructional unit was 
completed, each student took the first unit 
achievement test and an achievement score 
was collected. This process was repeated 
for the second instructional unit, with the 
only difference being the feedback delivery 
method that was implemented. After the 
second instructional unit test, the data 
collected were standardized on the same 
scale before analysis. The two tests had a 
different number of total questions, so the 
students’ scores on each of the unit tests 
were standardized based on a one 
hundred-point scale. Table 1 represents the 
experiment design of this study. 


Table 1 


Experimental Design 


Agriscience Applications 
Class Number 

Instructional Unit 1 : 
FFA History 

Instructional Unit 2: 
Leadership Development 

Class 1 : Daniele 

Traditional Instruction 

CPS Treatment 

Class 2: Joan 

CPS Treatment 

Traditional Instruction 

Class 3: Mike 

CPS Treatment 

Traditional Instruction 
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Population and Sample 

The population of this study was all 
students participating in agriscience 
applications courses at Southern Nash High 
School. Those students registering for the 
selected courses during the Fall 2004 
semester were automatically placed into the 
class group for which they registered. Three 
sections of Agriscience Applications were 
tested over a period of two instructional 
units. The intact groups were purposely 
assigned either a treatment or comparison 
group for instructional unit one and then the 
assignments were switched for instructional 
unit two. Sample sizes were determined by 
how many students enrolled in each section. 
Enrollment for classes 1, 2 and 3 were 23, 
20 and 18 respectively. The total sample size 
was 61 students. Each of the students 
experienced a lesson with both the audience 
response system feedback and the non- 
technology based feedback. Prior to the data 
analysis two groups were constructed with 
61 students’ achievement scores in each of 
the two groups. Group one was the 
treatment group or the student achievement 
scores from when the students participated 
in the audience response system feedback. 
Group two was the comparison group or 
when students participated in verbal and 
written response-contingent feedback. 

Instrumentation, Data Collection and 
Analysis 

The instruments in the quantitative phase 
of the study were instructional unit tests for 
each of the two instructional units that were 
created by a collaborative effort of the 
three instructors. The test items were 
drawn from a statewide test bank maintained 
by the department of public instruction. 
The tests were examined for both validity 
and reliability. A panel of experts 
verified content validity. To assess 
reliability, the Kuder-Richardson 
coefficient was used to determine 
coefficients of 0.79 for test one and 0.76 for 
test two. 

Data collection for this research 
occurred after the completion of 
instructional units one and two. The 
experimental process began with the 
teaching of instructional units one and two. 
This process included providing response- 


contingent verbal and written feedback 
using the same questions that were being 
presented in the treatment group. The 
feedback was provided to as many students, 
as the teacher determined feasible within the 
time constraints, usually only providing 
feedback to one or two students per 
question. Achievement scores from each 
student were collected for units one and two. 
Then the scores were combined to form the 
treatment group (group 1) and the 
comparison group (group 2). In the 
classroom using the audience response 
system, feedback was provided to all 
students, immediately with the aid of the 
audience response system. The results of 
daily questioning activities were not 
documented in the traditional feedback 
delivery classroom but were cataloged by 
the audience response system software. 
Teachers were allowed to use this data at 
their discretion. 

One achievement or test score was 
recorded from each student, in each of the 
two units. Achievement scores were 
collected based on the ratio of correct to 
incorrect answers. Then these ratios were 
converted into a 100-point scale to 
standardize aggregate scores for 
comparison. Descriptive statistics such as 
mean, mode, and median on this data were 
calculated. The instructional unit scores 
were combined into two groups, the 
treatment group and the comparison group 
and an independent samples t-test was used 
to analyze any differences present between 
the two groups. Every student participated in 
either the treatment group for instructional 
unit one or instructional unit two. 

Findings 

Table 2 represents the descriptive data 
collected from combining achievement 
scores calculated on a 100-point scale, into a 
comparison group and a treatment group. 
Group one in the table represents the 
treatment group and group two represents 
the comparison group. The treatment group 
or group one had a mean score of 89.98 on a 
100 point maximum scale, with a standard 
deviation (SD) of 8.817 and a standard error 
of the mean (SEM) of 1.116. The 
comparison group or Group two had a mean 
score of 84.41 on a 100 point maximum 
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scale, with a standard deviation (SD) 12.618 
and a standard error of the mean (SEM) of 


1.616. 


Table 2 


Achievement Scores by Group 


Group 


N 

M 

SD 

SE 

Treatment 

1 

61 

89.98 

8.71 

1.11 

Comparison 

2 

61 

84.41 

12.61 

1.61 


An independent samples t-test was 
conducted to test the null hypothesis 
for research question one. This test 
was conducted to compare group one, the 
treatment group and group two the 
comparison group and to examine any 
differences between the group 
achievement scores. As a preliminary 
requirement for conducting independent 
samples t-test, it is best to examine the 
variances of each group to determine if 
equal variances exist. The statistical 
test for homogeneity used in this research 


was Levene’s Test for equality of 
variances. 

Table 3 represents the results from the 
Levene’s Test for equality of variance. 
Levene’s Test for equality of variances 
determined that the data had an F = 7.335 
and a significance of p — 0.008, 

demonstrating that there is a significant 
difference in the variances of the two groups 
at the a = 0.05 level. As a result of these 
preliminary analyses, variances cannot be 
assumed to be equal when conducting the 
independent samples t-test calculation. 


Table 3 


Test of Homogeneity for the Variable of Achievement 


Levene Statistic 

dfl 

df2 

P 

7.335 

1 

120 

.008 


Table 4 represents the results of the 
independent samples t-test calculations, that 
compared the combined achievement scores 
from group one the treatment 
group and group two the comparison group. 
The results of the analysis of the variable 
of achievement were analyzed at the 
a = 0.05 level and due to the preliminary 
analysis of Levene’s Test for equality of 
variance, the variances can not be 


assumed to be equal. The independent 
samples t-test revealed that the 
variable of achievement has a score of t = 
2.835, with degrees of freedom of 107. The 
calculations also produced data resulting in 
a two tailed significance of p = 0.005, a 
mean difference of 5.567 with a standard 
error of proportion of 1.964, and a 95% 
Confidence Interval of the difference of 
1.674. 
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Table 4 


Comparison of Achievement Mean Scores 


Achievement 

t 

df 

P 

Mean 

Difference 

Std. Error 
Difference 

95% Confidence 
Interval 







Lower 

Upper 

Equal variances 
assumed 

2.835 

120 

.005 

5.567 

1.964 

1.679 

9.455 

Equal variances 
not assumed 

2.835 

107 

.005 

5.567 

1.964 

1.674 

9.460 


Conclusion and Recommendations 

This study sought to determine if 
differences in student achievement existed 
between groups of agriscience applications 
students who used an audience response 
system and those who used more traditional 
written and verbal response-contingent 
feedback methods. It is concluded that 
students receiving feedback through an 
audience response system have higher 
achievement scores than those who received 
feedback through non-technology based 
methods. Based upon the findings of this 
research the null hypothesis was rejected. 

The implementation of an audience 
response system in the agriscience 
classroom environment resulted in a 
statistically significant difference in student 
achievement, when compared to students 
receiving the more traditional written and 
verbal response contingent feedback 
methods. Student achievement improved 
when using an audience response system as 
a primary feedback method. The conclusion 
of this study also reflected in earlier studies 
conducted by Abrahamson (1998) and 
Everett and Ranker (2002). Both of these 
studies found similar positive attitudes from 
students and teachers related to the use of an 
audience response system. Similar negative, 
teacher perceived impacts were discovered 
by Everett and Ranker including the 
perception that using an audience response 
system was more time consuming than 
providing written and verbal feedback. 

Recommendations 

This research recommends that the 
audience response system could be 


integrated into agriscience applications 
classrooms with some degree of confidence 
of success. The availability of funding, 
training time, preparation time, and 
implementation time should be considered 
when implementing the audience response 
system. Furthermore, this study was 
conducted with a relatively small sample 
and under a particular set of circumstances. 
Generalizations regarding the efficacy of 
audience response systems in the classroom 
requires further substantiation through 
additional research and replications in a 
variety of classroom environments. Finally, 
testing and additional research should be 
done with new technologies as they emerge. 
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